Adding time_spine field for manifest #112

pgoslatara · 2024-09-18T16:36:10Z

User description

The latest version of manifest.json from dbt Cloud contains a time_spine field for nodes. This PR adds this field so manifests can be parsed.

Extra info: this is what the metadata field looks like from the manifest.json I downloaded:

{
    "metadata": {
        "dbt_schema_version": "https://schemas.getdbt.com/dbt/manifest/v12.json",
        "dbt_version": "2024.9.239",
        ...
    },

PR Type

enhancement

Description

Added a new time_spine field to the Nodes4 class in the manifest_v12.py file.
This field is optional and allows for parsing the time_spine attribute from the latest manifest.json version.

Changes walkthrough 📝

Relevant files

Enhancement

manifest_v12.py Add `time_spine` field to `Nodes4` class in manifest parser dbt_artifacts_parser/parsers/manifest/manifest_v12.py Added `time_spine` field to `Nodes4` class. The `time_spine` field is optional and of type `str`.	+1/-0

💡 PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

coderabbitai · 2024-09-18T16:36:20Z

Warning

Rate limit exceeded

@pgoslatara has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 14 minutes and 12 seconds before requesting another review.

How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

Commits

Files that changed from the base of the PR and between 7f094d4 and b15a913.

Walkthrough

The changes introduce a new optional attribute, time_spine, to the Nodes4 class in the dbt_artifacts_parser/parsers/manifest/manifest_v12.py file. This attribute is designed to hold a string value, allowing the class to potentially manage or represent temporal data. The modification enhances the existing structure of the class by adding this new property.

Changes

File Path	Change Summary
dbt_artifacts_parser/parsers/manifest/manifest_v12.py	Added optional attribute `time_spine: Optional[str] = None` to `Nodes4` class.

Sequence Diagram(s)

Poem

In the land of code where rabbits play,
A new attribute hops in today.
With time_spine added, oh what a sight,
Managing time with all of its might!
So let’s celebrate this change with cheer,
For temporal magic is finally here! 🐇✨

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
-- I pushed a fix in commit <commit_id>, please review it.
-- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
-- @coderabbitai generate unit testing code for this file.
-- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
-- @coderabbitai generate interesting stats about this repository and render them as a table.
-- @coderabbitai read src/utils.ts and generate unit testing code.
-- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
-- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

codiumai-pr-agent-pro · 2024-09-18T16:36:26Z

PR Reviewer Guide 🔍

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ No key issues to review

codiumai-pr-agent-pro · 2024-09-18T16:36:40Z

PR Code Suggestions ✨

Category	Suggestion	Score
Data validation	✅ Add a validator for the time_spine field to ensure data integrity Suggestion Impact: The suggestion to add a validator for the time_spine field was implemented by changing the type of time_spine from Optional[str] to Optional[TimeSpine], which likely includes validation logic within the TimeSpine class. code diff: - time_spine: Optional[str] = None + time_spine: Optional[TimeSpine] = None Consider adding a validator for the `time_spine` field to ensure it follows a specific format or meets certain criteria. This can help prevent invalid data from being assigned to this field. dbt_artifacts_parser/parsers/manifest/manifest_v12.py [792] -time_spine: Optional[str] = None +time_spine: Optional[str] = Field(None, validator=validate_time_spine) Apply this suggestion Suggestion importance[1-10]: 8 Why: Adding a validator for the `time_spine` field is a good practice to ensure that the data assigned to it meets specific criteria or format, which enhances data integrity and prevents potential errors.	8
Documentation	Add a description to the time_spine field for better documentation Consider adding a description for the `time_spine` field using the `Field` constructor. This can provide more context about the purpose and expected format of the field. dbt_artifacts_parser/parsers/manifest/manifest_v12.py [792] -time_spine: Optional[str] = None +time_spine: Optional[str] = Field(None, description="Specifies the time spine for the SQL operation") Apply this suggestion Suggestion importance[1-10]: 7 Why: Including a description for the `time_spine` field improves code documentation, providing context and clarity on its purpose and expected format, which is beneficial for future maintainability and understanding.	7

💡 Need additional feedback ? start a PR chat

pgoslatara · 2024-09-18T16:42:02Z

Ahh, just cam across this, looks like time_spine is not a simple string.

pgoslatara · 2024-09-18T16:52:21Z

Updated!

yu-iskw · 2024-09-19T00:21:58Z

@pgoslatara Thank you for the contribution. But, I think supporting the feature is a little early, as it will be released as a part of dbt-core 1.9.

We don't change the pydantic class by hand, as we generate pydantic classess from the JSON schemas of dbt-core.
- https://github.com/dbt-labs/dbt-core/tree/main/schemas/dbt
According to my research, the property was added as a part of dbt 1.9 which isn't released yet. I want to support stable releases to dbt-core versions, as properties which aren't released yet are potentially changed until the release. So, I want to update the pydantic class to manifest v12 after dbt-core 1.9 is released.

pgoslatara · 2024-09-19T07:36:52Z

Thanks @yu-iskw!

What does this mean for supporting dbt Cloud? There is now a versionless option, in fact this will soon become the only option (doc) and is currently the recommended option (doc). Does this mean dbt-artifacts-parser will be incompatible with versionless dbt Cloud artifacts?

yu-iskw · 2024-09-19T08:16:15Z

@pgoslatara

This project mainly supports dbt Core, because the JSON schema for dbt Cloud aren't available in public. For instance, as far as I know, semantic_manifest.json isn't available. So, we can't fully support dbt Cloud right now. So, we can't generate a pydantic class for that. Basically, we can generate pydantic classes based on JSON schemas of dbt Core.

Meanwhile, we have evolved dbt artifacts of dbt Core as upgrading the version like manifest v10, v11 and v 12. If we upgrade the versions of artifacts in the future, generated artifacts in dbt Cloud have to follow the versions as well regardless versionless. Each artifacts has a property about a dbt version. As the stable version of dbt Core 1.9 isn't released yet, it might be early to update manifest v12 based on dbt Core 1.9.

https://github.com/dbt-labs/dbt-core/blob/3308a4365ed7a0aebea602b4444db2b08552ef37/schemas/dbt/manifest/v12.json#L14-L17

Adding time_spine field for manifest

6bdaf08

pgoslatara requested a review from yu-iskw as a code owner September 18, 2024 16:36

codiumai-pr-agent-pro bot added the enhancement New feature or request label Sep 18, 2024

codiumai-pr-agent-pro bot added the Review effort [1-5]: 1 label Sep 18, 2024

pgoslatara mentioned this pull request Sep 18, 2024

Missing fields in constraints model for manifest v12 #109

Open

pgoslatara force-pushed the model-time-spine branch from 7f094d4 to deda5ec Compare September 18, 2024 16:47

Not a simple string silly

b2c708f

pgoslatara force-pushed the model-time-spine branch from deda5ec to b2c708f Compare September 18, 2024 16:50

Merge branch 'main' into model-time-spine

b15a913

pgoslatara mentioned this pull request Sep 19, 2024

[Bug] dbt Cloud manifest is not parsable godatadriven/dbt-bouncer#240

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding time_spine field for manifest #112

Adding time_spine field for manifest #112

pgoslatara commented Sep 18, 2024 •

edited by codiumai-pr-agent-pro bot

Loading

coderabbitai bot commented Sep 18, 2024 •

edited

Loading

Rate limit exceeded

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

codiumai-pr-agent-pro bot commented Sep 18, 2024

codiumai-pr-agent-pro bot commented Sep 18, 2024 •

edited

Loading

pgoslatara commented Sep 18, 2024

pgoslatara commented Sep 18, 2024

yu-iskw commented Sep 19, 2024

pgoslatara commented Sep 19, 2024

yu-iskw commented Sep 19, 2024

Adding time_spine field for manifest #112

Are you sure you want to change the base?

Adding time_spine field for manifest #112

Conversation

pgoslatara commented Sep 18, 2024 • edited by codiumai-pr-agent-pro bot Loading

User description

PR Type

Description

Changes walkthrough 📝

coderabbitai bot commented Sep 18, 2024 • edited Loading

Rate limit exceeded

Walkthrough

Changes

Sequence Diagram(s)

Poem

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

codiumai-pr-agent-pro bot commented Sep 18, 2024

PR Reviewer Guide 🔍

codiumai-pr-agent-pro bot commented Sep 18, 2024 • edited Loading

PR Code Suggestions ✨

pgoslatara commented Sep 18, 2024

pgoslatara commented Sep 18, 2024

yu-iskw commented Sep 19, 2024

pgoslatara commented Sep 19, 2024

yu-iskw commented Sep 19, 2024

pgoslatara commented Sep 18, 2024 •

edited by codiumai-pr-agent-pro bot

Loading

coderabbitai bot commented Sep 18, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

codiumai-pr-agent-pro bot commented Sep 18, 2024 •

edited

Loading